Object detection in crowded scenes using context-driven label propagation

ABSTRACT

A computer implemented method for detecting an object in a crowded scene utilizing an image capturing device. The method includes receiving an image of a predetermined area. From the image, the existence of selected portions as representing an entity of a selected class is determined. Each selected portion is assigned an initial confidence value that the selected portion is an entity representative of a selected class. Each selected portion is evaluated with each other to determine a context confidence value. The context confidence value and initial confidence value are utilized to determine which of the one or more selected portions are entities of a selected class.

BACKGROUND

The present disclosure relates to image processing for detectingselected objects in an image, and more particularly to the utilizationof context-driven label propagation to identify groups of selectedobjects in crowded and/or cluttered scenes.

The need to detect and/or identify selected objects such as people orvehicles in images is central to a wide range of applications such asvideo surveillance and autonomous driving. Object detection forindividuals and vehicles has been utilized in a variety of systems,including, for example, computer vision, deformable part models (DPM),poselets and deep learning. Object detection is particularly challengingwhen the objects in a given scene are at least partially obstructed orunclear because of a cluttered background, or because of the largenumber of potential pose variations.

SUMMARY

In accordance with an embodiment, a method for detecting objects in acrowded scene is provided. The method includes receiving, using aprocessor system, image data representing objects in a predeterminedarea. The processor system makes an initial determination that selectedportions of the image data represent one or more entities of apredetermined class of object. An initial confidence value of theselected portions of the image data is determined wherein the initialconfidence value identifies likelihood that the selected portions of theimage data represent one or more of a selected class of object. Acontext confidence value of the selected portions of the image data isdetermined wherein the context confidence value identifies a contextualrelationship between the selected portions of the image data. Thecontext confidence value and the initial confidence value are utilizedto determine that one of the selected portions of the image datarepresents an entity representative of a selected class of object.

In accordance with a further embodiment, a computer program product fordetecting an object representative of a predetermined class of object ina crowded scene wherein the computer program product includes a computerreadable storage medium having program instructions embodied therewith,and wherein the computer readable storage medium is not a transitorysignal. The program instructions are readable by a processing circuit tocause the processing circuit to perform a method. The method includesreceiving image data representing objects in a predetermined area andmaking an initial determination that selected portions of the image datarepresent one or more entities of a selected class of object. The methodfurther includes determining an initial confidence value of the selectedportions of the image data wherein the initial confidence valueidentifies a likelihood that the selected portions of the image datarepresent one or more entities of a selected class of object andutilizing the context confidence value and the initial confidence valueto determine that one of the selected portions of the image datarepresents an entity representative of a selected class of an object.

In another embodiment of the invention, an individual identificationsystem for detecting objects in a crowded scene utilizing an imagecapturing device is provided. The system includes a processor incommunication with one or more types of memory and wherein the processoris configured to receive image data representing objects of apredetermined area. The processor makes an initial determination thatselected portions of the image data represent one or more entities of apredetermined class of object. An initial confidence value of theselected portions of the image data is determined by the processorwherein the initial confidence value identifies a likelihood that theselected portions of the image data represents one or more entities of aselected class of object. A context confidence value of the selectedportions of the image data is also determined by the processor whereinthe context confidence value identifies a contextual relationshipbetween the selected portions of the image data. The context confidencevalue and initial confidence value are utilized to determine that one ofthe selected portions of the image data represents an entityrepresentative of a selected class of object.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter which is regarded as the invention is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The forgoing and other features, and advantages ofthe invention are apparent from the following detailed description takenin conjunction with the accompanying drawings in which:

FIG. 1 illustrates the capturing of an image and processing fordetermining the presence of an object according to one or moreembodiments of the present disclosure;

FIG. 2 illustrates the creation of a context graph via context labelpropagation according to one or more embodiments of the presentdisclosure;

FIG. 3 is a block diagram illustrating one example of a processingsystem according to one or more embodiments of the present disclosure;

FIG. 4 is a flow diagram of a method of detecting objects in a crowdedscene in accordance with one or more embodiments of the presentdisclosure;

FIG. 5 is a flow diagram of the processing of context informationaccording to one or more embodiments of the present disclosure;

FIG. 6 is an illustration of a social context assessment in accordancewith one or more embodiments of the present disclosure;

FIGS. 7a, 7b and 7c illustrate multiple iterations of contextinformation for detecting individuals in a crowded scene in accordancewith one or more embodiments of the present disclosure; and

FIG. 8 is an illustration of detections based upon multiple iterationsof context information in accordance with one or more embodiments of thepresent disclosure.

DETAILED DESCRIPTION

Embodiments of the present disclosure present a novel image detectionsystem for recognizing an object which represents an entity of apredetermined class of objects in crowded scenes which includesidentifying the object and its contextual relationship with individuals.In exemplary embodiments, methods and system for detecting objects suchas individuals in an image are provided. The systems and methods alsoinclude determining an object's context and modeling interactionsbetween other detected objects. In a crowded scene, people usually formgroups where they interact with each other, both spatially and socially.A group of people, queuing, sitting or walking together, indicatespatial closeness and similar scales. There also exists strong socialpatterns in the group such as facing (i.e. two people face each other)and following (e.g. people stand or sit side by side). In exemplaryembodiments, methods and systems for identifying contextual cues andcreating models of individuals' interactions are provided to determineif an identified object in an image is representative of an individual.

Turning now to an overview of the present disclosure, one or moreembodiments provide methods, systems and computer program products fordetecting objects of a predetermined class in crowded scenes by usingcontext driven label propagation are provided. In exemplary embodiments,an image of a predetermined area is taken by an image capturing devicesuch as a camera or video recorder. Image data representing objects isreceived and selected portions of the image data are evaluated formaking an initial determination that the selected portions represent oneor more entities of a selected class of objects. The selected portionsare evaluated utilizing imaging techniques which include recognition ofcommon components of an object of a selected class. For instance, if thedesired selected class includes an individual, then facial recognitionand body recognition features would be analyzed. Such techniques includedeformable part models (DPM), poselets and deep learning. An additionalclass of object maybe a vehicle wherein recognition of a vehicle bodyincluding tires would be analyzed. For the selected portions which meeta certain threshold of having recognizable features representing anentity representative of a selected class, an initial confidence valueis determined. For instance, as shown in FIGS. 1 and 2, those entitieswhich appear to possibly represent individuals are given a score rangingfrom 2 to 10. In this situation, the value of 10 is the highestconfidence level that the image captured is indeed an individual basedupon classic imaging processing techniques. However, due to certainimages being unclear, the initial confidence value of certain selectedportions of image data are lower. The threshold may vary depending onthe particular environment involved. For instance, in an autonomousvehicle driving scenario, the inclusion of false positives is morebeneficial than incorrectly not identifying an individual or anothervehicle based upon an unclear image hence the threshold for identifyingan identified object based on visual recognition is low. However, in astadium setting where the occurrence of individuals is expected, thethreshold for identifying an identified object based on visualrecognition could be higher. The establishment of a baseline asdescribed is based solely on visual recognition techniques. Such atechnique removes stationary objects such as trash cans and the like,but may include an object such as a statue as a false positive.

With the establishment of a baseline, the method further includesevaluating the initial confidence score by comparing the selectedportion of an image with additional selected portions of the image viacontextual relationships. In the exemplary embodiment, the relativecontextual relationships include spatial, scale, social, and overlap.These contextual relationships are each individually evaluated withrespect to an attraction attribute or a repulsion attribute. Eachcontextual relationship with a respective selected portion of the imageis evaluated with respect to additional selected portions within apredetermined range. For example, the scale context of a particularselected portion may be evaluated with respect to those selectedportions within a ten foot area, or the social context may be evaluatedwith respect to those selected portions within a five foot area. Eachrelationship is aggregated to produce a context confidence value. If thecontext confidence value is greater than a predetermined threshold, thenthe initial confidence value that a selected portion is an entityrepresentative of a selected class may be validated. If the contextconfidence value is less than a predetermined threshold, then theinitial confidence value that an identified object is an entity of arepresentative class may be seen as a false positive.

The invention focuses on post image processing to determine the presenceof an entity representative of a selected class based on contextualenvironmental clues where such objects would not have been considered anentity representative of a selected class from prior image processingtechniques due to the quality of the initial image. An embodiment of thepresent invention further processesvisual images for identifying anobject representative of a class such as individuals via an imagedetector which when processed under known image processing techniqueswould fail to identify selected portions of an image as representing anindividual. FIG. 1 illustrates a group of people in a social interactionsetting as recorded by a surveillance camera. In this embodiment, theselected class of objects would include individuals. In otherembodiments, the selected class of objects may include vehicles. Asshown in FIG. 1, people are grouped with the two people on the leftfacing each other and two people on the right following or sitting sideby side. Initial detections are identified as either high confidence orlow confidence. As seen in FIG. 1, an initial processing of an image 200taken by an image capturing device 302 such as a surveillance cameraidentifies several selected portions. Each selected portion is processedfor determining the likelihood that the selected portion is anindividual. As shown in FIG. 1, selected portions 102-118 are identifiedas potential individuals and assigned initial confidence values basedupon image processing characteristics utilized for recognizingindividuals such as facial and body recognition techniques which areknown to those skilled in the art such as deformable part models (DPM),poselets and deep learning.

The relationship of the individuals is presented in a context graphillustrated in FIG. 2. A contextual analysis between the individuals isundertaken. A bold edge indicates a mutual attraction between two nodes,i.e. they are contextually compatible. Oppositely a dotted edge suggestsan opposite relationship, i.e. repulsion. The framework applies labelpropagation to boost up weak detections while suppressing irrelevantfalse alarms.

As shown in FIG. 2, the context graph 350 captures the interactionstrength between selected portions or detections. Each node in FIG. 2represents a selected portion from FIG. 1. As shown in FIG. 1 and FIG.2, the selected portion/node relationship is as follows:

Selected Portion Node 102 220 104 225 106 230 108 240 110 235 112 245114 250 116 255 118 260

A context graph that incorporates contextual information available fromthe scene is built. The contextual information can include spatial,scale, social, and overlap cues. This contextual information is utilizedfor label propagation. The label propagation framework which builds thecontext graph spreads both positive and negative contextual potentialsalong edges depending on contextual attraction or repulsion; as aresult, contextually compatible selected portions representing entitiesrepresentative of a selected class such as an individual gets reinforcedby receiving positive potentials during the propagation while falsealarms are contained due to being negated by their incompatibility withtrue detections. This approach is illustrated in FIG. 2 whichillustrates an initial scene where true “labeled” data has yet beendeveloped and the initial context graph includes greedy-like inferenceswhich iteratively adds the best identified object with the mostpotential gain as a label at each time to initialize a new round ofpropagation. The process repeats until convergence when no newidentified object can be labeled.

As seen in FIG. 2, in the exemplary embodiment, different selectedportions may initially have an initial positive confidence value basedupon a positive visual recognition. However, upon processing, thecontext confidence value may decrease this value. In this scenario theselected portion would be deemed a false alarm. However, in otherscenarios, an initial identified object may have their initialconfidence value increased or remain the same, in this scenario theinitial confidence value identifying that the selected portion is anindividual is validated.

As illustrated in FIG. 2, a respective selected portion is evaluatedwith respect to multiple surrounding selected portions in order toprocess the context graph. The Context Graph 350 illustrates the overallconfidence that a selected portion is an individual. The selectedportions from FIG. 1 are identified as nodes having a respective initialconfidence value. By evaluating the contextual relationships of theselected portions, a context confidence value is established. These twometrics are combined to produce an overall confidence value that theselected portion is representative of an individual. The following tablereflects the attributes identified in Context Graph 350.

Initial Context Overall Node Confidence Value Confidence ValueConfidence 220 3 −6 −3 225 10 0 10 230 1.5 0 1.5 235 2 −6 −4 240 5 2 7245 5 −3 −2.5 250 2.5 1.5 4 255 2 −4 −2 260 1.5 1 2.5

The building of the contextual graph involves several steps. In anexemplary embodiment, given an image, let X={x_(i), i=1:m} be a set of mselected portions generated by an underlying people detector such asvisual recognition instructions. The detection threshold is variable andin some exemplary embodiments is purposely set to a low detectionthreshold to allow for more true detections in X, which unfortunatelyalso gets many more undesirable false alarms. Therefore, the task is tofind a subset of X that covers as many as possible true detections,meanwhile with the fewest false alarms brought in. Mathematically, anindicator vector is sought where Y={y₁, y₂, . . . , y_(m)}Tε{0, 1} (1means true people detection otherwise background) that maximizes apotential function Ψ(X, Y), such that the visual detections agree on thecontextual setting of the image. Equation 1 defines the potentialfunction as follows:

${\Psi\left( {{??},Y} \right)} = {{\sum\limits_{i = 1}^{m}\;{y_{i}{\psi^{u}\left( x_{i} \right)}}} + {\alpha{\sum\limits_{i = 1}^{m}\;{y_{i}{\psi^{c}\left( {x_{i},{??},Y} \right)}}}}}$

where ψ^(u)(•) is the unary potential that utilizes the originaldetection score of a selected portion. ψ^(c)(•) represents the totalcontextual potentials of an object received from others. α is a constantnumber balancing these two terms. A model of the contextual interactionbetween any two selected portions is developed via a context graph G.Thus producing equation 2 ψ^(c)(x_(i),

, Y)

(x_(i), Y) where ψ^(G) (x_(i), X, Y) measures how much contextualpotential identified object x_(i) can be obtained from validatedselected portions (reflected by Y) based on G.

A context graph is created that is an undirected graph G=(V, E) used forlabel propagation where V corresponds to a set of selected portions andE indicates the strength of contextual interactions between each pair ofselected portions. While the focus is to reward selected portions whichare contextually consistent to true detections, suppressing false alarmsis equally important during label propagation as the input initiallycontains substantial errors. Accordingly, two types of strengths areconsidered; attraction e+ and a repulsion e− when constructing thecontextual graph G. Attraction measures contextual compatibility betweentwo identified entities while repulsion relates to contextualinconsistencies. In an exemplary embodiment, four types of contextualcues are utilized. These are namely scale, spatial, overlap and socialof which their attraction strengths are denoted as e+ scale, e+ spatial,e+ overlap, and e+ social. Similarly three types of cues are utilized inevaluation repulsion. These include scale, spatial and overlap withtheir respective repulsion strengths denoted as e-scale, e-spatial ande-overlap.

The respective context features are mapped to a number between 0 and 1to indicate the strength of the contextual interaction. Spatial contextis explored by the image distance d (x_(i), x_(j)) between two selectedportions. To eliminate the effects of image resolution and cameraperspective, d (xi, xj) is further normalized as follows:

${f_{sp}\left( {x_{i},x_{j}} \right)} = {\frac{2\;{d\left( {x_{i}x_{j}} \right)}}{h_{i} + h_{j}} \cdot {{\min\left( {\frac{h_{i}}{h_{j}},\frac{h_{j}}{h_{i}}} \right)}.}}$

where h_(i) and h_(j) are the image heights of the selected portionsx_(i) and x_(j). The ratio h_(i)/h_(j) compensates camera perspective asthis ratio can reflect the depth change of the two selected portions.The minimum operation is utilized to obtain symmetric feature. The sumh_(i) and h_(j) further normalizes the distance into the unit of humanheight. Scale context is evaluated by the physical height ratio of twoselected portions. Initially an estimate is made of the image locationof a horizon line v_(o). By assuming that all selected portions are allgrounded and upright, the physical height of identified object x_(i) isdefined as follows:ζ_(i) =h _(i)ζ_(c) /v _(i) −v _(o)

Where h_(i) and v_(i) encode the physical height and image locator ofx_(i) respectively. ζc is the camera height. Consequently, given any twodetections, a value pair is obtained (v|₀(h_(i)−h_(j)),h_(i)v_(j)−h_(j)v_(j)).

With multiple (≧3) strong detections (i.e. high confidence), v₀ caneasily be estimated by least square fitting. The physical height ratioof two selected portions can then be defined as:

${f_{sc}\left( {x_{i},x_{j}} \right)} = {\min\left( {\frac{h_{i}\left( {v_{j} - v_{o}} \right)}{h_{j}\left( {v_{i} - v_{0}} \right)},\frac{h_{j}\left( {v_{i} - v_{o}} \right)}{h_{i}\left( {v_{j} - v_{0}} \right)}} \right)}$

The minimum is used for feature symmetry. In operation, if the height ofa standing selected portion is 1, a sitting identified object isregarded as 0.75 height.

Social context is an attribute which reflects social interactionsbetween two detections measured in virtue of pose and body (or head)orientation. A pose classifier (standing or sitting) was designated asRBF-kernel SVM with 1200-dim poselet activation vector (PAV). The 2-dimprobability output p of this classifier is then used to evaluate posesimilarity. Body orientations indicate the position of co-existing truedetections. In the exemplary embodiment, two orientation patterns aremodeled, namely following and facing.

FIG. 6 illustrates the concepts of following and facing. A small ∠θ,i.e. included angle of body orientations, indicates a following pattern,while ∠θ₁ and ∠θ₂ when both being small, exhibit a facing pattern. Thestrongest orientation pattern is used, thus producing∫so=min(∠θ,max(∠θ1 and ∠θ2))*|p ₁ −p ₂|₂.

Overlap context utilizes the overlap ratio of two bounding boxes toexpress location compatibility of two selected portions, i.e.,f_(ov)=(B_(i)∩B_(j)/(B_(i)∪B_(j)), where B_(i) and B₃ are bounding boxesof x_(i) and x_(j), respectively. If the ratio is of a certainthreshold, then it is determined that the selected portions are actuallyindicative of only one individual.

A data driven approach is utilized to learn a mapping function F: f→efor each of the contextual pattern, using 120 images from StructuredGroup Dataset SGD that are independent from the evaluation subset.Gaussian kernels are utilized to model mapping functions F. Parametersof these Gaussian functions are estimated by fitting the distributionsof the respective context patterns. Spatial attraction e_(sp)+ ismodeled as a 2-component Gaussian kernel, which corresponds to a maximuminfluence around 0.3 human height and vanishes after 1.5. e_(sp) ⁻=1when F_(sp)<0.1. For e_(sc)+, a Gaussian function in the range of |0,0.8| is fit. To counteract height differences of individuals and milderrors of bounding boxes, e_(sc)+ is set to =1 when F_(sc)ε|0.8, 1|,e_(ov)+, e_(ov) ⁻, and e_(so)+ are done in a similar way.

The scale and overlap contexts are deductive patterns, i.e. they arediscriminative to tell whether an identified object is true or not. Forinstance, if a identified object goes against true detections withregard to scale (small e_(sc)+), then a strong repulsion should be given(large e_(sc) ⁻). Thus e_(sc)−=1−e_(sc)+ and e_(ov) ⁻=1−e_(ov)+. Inopposite, spatial and social cues are not deductive as it cannot beinferred whether or not a hypotheses is invalid, even if it is remotefrom true detections or no social interactions are observed.

Given a context graph G, the contextual potential)

(x_(i), Y) (iε[1:

) is obtained. An initial potential vector ZεR^(m) is initialized as Ywhere YεR^(m) for m selected portions. If the contextual potential of aselected portion x_(i) is targeted z_(i) is set to =0 to avoidself-reinforcement. Under such a setting, a selected portion x_(i) islabeled when y_(i)≠0 and unlabeled otherwise. As aforementioned, strongtrue detections is more robust in propagating potential, therefore G isreweighted. Logistic regression is applied to normalize unary

(x_(j))(jε[1:m]) into w_(j)ε(0, 1) which can be regarded as the truedetection probability of selected portion x_(j); w_(j) is set to =1 ify_(j)=1, since y_(j)=1 means validated true detection. Then each columnof G is re-weighted by w_(j), i.e., G′.j=_(wj), G.j. Matrix G′ isfurther row-normalized such that

=

/Σ_(k)

|, which is critical for the convergence of the propagation algorithm.The potential propagation algorithm in Algorithm. 1 is as follows:

Algorithm 1 Potential propagation for ψ^(G) (x_(i),Y)  1: Input: givenG, Y.  2: Output: ψ^(G) (x_(i), Y)  3: Initialize Z = Y, z_(i) = 0  4:Obtain G by re-weighting and row-normalization  5: while Z does notconverge do  6: Z ← GZ  7: Z = max (Z, 0)  8: ∀ _(j) ∈ [1 : m], j ≠ i,if yj = 1, zj = 1,  9: end while 10: ψ^(G)(x_(i), Y) ← GZ

In line 6, potential is progagated based on G and the potential vector Zin previous iteration. In line 7, all negative elements are reset inpotential vector Z back to zero in order to cut off the potentialdefused by false alarms as the identification of an object as anindividual is a true detection or a false alarm. In line 8, elements arereplenished with the initial value 1. The propagation algorithm repeatsfrom line 6 to line 8 until Z coverages. One more propagation isexecuted in line 10, in order to output the ψ^(G) (x_(i), Y) with bothpositive and negative contextual potential.

Considering contextual potential ψ^(G) (x_(i), Y), optimizing theobjective function in Equation 1 is NP hard. The proposed potentialpropagation algorithm is combined with a forward search, aiming at asub-optimal Ŷ=max_(y) ψ(X, Y). Different from conventional graph-basedpropagation with fixed “labeled” instances, unconfirmed selectedportions are progressively validated and propagate their potential innext run. Let S′={i|y_(i)=1, iε[1:m]} denote the confirmed set ofselected portions at t iteration. The potential change by instancingselected portions x_(i) is defined as follows:

${\Delta\left( x_{i} \right)} = {{{\Psi\left( {{??},{Y\left( {S^{t}\bigcup i} \right)}} \right)} - {\Psi\left( {{??},{Y\left( S^{t} \right)}} \right)}} = {{\psi^{u}\left( x_{i} \right)} + {\alpha\left( {{\psi^{??}\left( {x_{i},{Y\left( S^{t} \right)}} \right)} + {\sum\limits_{j \in S^{t}}\;{\psi^{??}\left( {x_{j},{1(i)}} \right)}}} \right)}}}$where ψ^(u)(x_(i)) is the unary potential: ψ^(G) (x_(i), Y (S^(t)))measures the contextual potential identified object x_(i) obtains frominstanced identified object in S^(t); Y (St) is the label vector thaty_(k)=1, if kεS^(t) and 0 otherwise; ΣjεS^(t)ψg (x_(j), 1(i)) representspotentials that identified object imposes onto instances(s) in S^(t),where 1 (i) is an indicator vector with only ith element equals 1 andothers are 0s. The progressive inference algorithm is as follows:

Algorithm 2 Progressive inference for Ŷ, Ŝ  1: Input: given G, X.  2:Output: Ŷ, Ŝ  3: Initialize Y = 0 and S^(o)= Ø  4: First instance i* =arg maxi ψ^(u) (x_(i)), set S¹ = {i*}.  5: while Δ(x_(i)) > 0 do  6: ∀i∈ [1 : m], y_(i) ≠1, i* = arg max_(i) Δ (x_(i),)(Alg.1)  7: Update Y:y_(i*) = 1 i*  8: S^(t+1) ← S^(t) ∪  9: t + 1 ← t 10:  end while 11: Ŷ,← Y, Ŝ ← S^(t)

The progressive inference algorithm starts with an empty set S and azero vector Y. The first selected portion is selected according to unarypotential only. During each iteration, one unconfirmed selected portionis evaluated with the largest potential change Δ(x_(i)) and updatesS^(t) and Y accordingly. Algorithm 2 runs line 6 to 9 repetitively andinstance one selected portion in each iteration, until adding any otherdetections could not enhance the total potential ψ(X, Y). Obviously, bygrowing S^(t) alternatively, contextual potentials from true detectionsare progressively propagated. When the algorithm ceases, the selectedportions in Ŝ are regarded as true detections while others are declaredfalse alarms. All detections are summed by their unary and contextualpotentials, i.e.,

${\psi^{\prime}\left( x_{i} \right)} = {{\psi^{u}\left( x_{i} \right)} + {\alpha\left( {{\psi\;{G\left( {{xi},\hat{Y},} \right)}} + \left( {\sum\limits_{j \in {St}}{\psi\;{G\left( {{xj},{1(i)}} \right)}}} \right)} \right.}}$

where ψ′(x_(i)) could be positive or negative. After rescoring, value 0is further used as the cutoff threshold to differentiate true detectionsand false alarms.

Referring to FIG. 3, there is shown an embodiment of a processing system100 for implementing the teachings herein. In this embodiment, thesystem 100 has one or more central processing units (processors) 101 a,101 b, 101 c, etc. (collectively or generically referred to asprocessor(s) 101). In one embodiment, each processor 101 may include areduced instruction set computer (RISC) microprocessor. Processors 101are coupled to system memory 114 and various other components via asystem bus 113. Read only memory (ROM) 102 is coupled to the system bus113 and may include a basic input/output system (BIOS), which controlscertain basic functions of system 100.

FIG. 3 further depicts an input/output (I/O) adapter 107 and a networkadapter 106 coupled to the system bus 113. I/O adapter 107 may be asmall computer system interface (SCSI) adapter that communicates with ahard disk 103 and/or tape storage drive 105 or any other similarcomponent. I/O adapter 107, hard disk 103, and tape storage device 105are collectively referred to herein as mass storage 104. Operatingsystem 120 for execution on the processing system 100 may be stored inmass storage 104. A network adapter 106 interconnects bus 113 with anoutside network 116 enabling data processing system 100 to communicatewith other such systems. A screen (e.g., a display monitor) 115 isconnected to system bus 113 by display adaptor 112, which may include agraphics adapter to improve the performance of graphics intensiveapplications and a video controller. In one embodiment, adapters 107,106, and 112 may be connected to one or more I/O busses that areconnected to system bus 113 via an intermediate bus bridge (not shown).Suitable I/O buses for connecting peripheral devices such as hard diskcontrollers, network adapters, and graphics adapters typically includecommon protocols, such as the Peripheral Component Interconnect (PCI).Additional input/output devices are shown as connected to system bus 113via user interface adapter 108 and display adapter 112. A keyboard 109,mouse 110, and speaker 111 all interconnected to bus 113 via userinterface adapter 108, which may include, for example, a Super I/O chipintegrating multiple device adapters into a single integrated circuit.

In exemplary embodiments, the processing system 100 includes a graphicsprocessing unit 130. Graphics processing unit 130 is a specializedelectronic circuit designed to manipulate and alter memory to acceleratethe creation of images in a frame buffer intended for output to adisplay. In general, graphics processing unit 130 is very efficient atmanipulating computer graphics and image processing, and has a highlyparallel structure that makes it more effective than general-purposeCPUs for algorithms where processing of large blocks of data is done inparallel.

Thus, as configured in FIG. 3, the system 100 includes processingcapability in the form of processors 101, storage capability includingsystem memory 114 and mass storage 104, input means such as keyboard 109and mouse 110, and output capability including speaker 111 and display115. In one embodiment, a portion of system memory 114 and mass storage104 collectively store an operating system to coordinate the functionsof the various components shown in FIG. 3.

As seen in FIG. 4, in the exemplary method, an image is captured andprocessed at block 400. Once captured, the image is processed by aprocessor utilizing visual recognition parameters to determine if aselected portion is potentially representative of an entity of apredetermined class of an object. The class may include a vehicle or anindividual or any object which may be classified via image processing.In the embodiment shown in FIG. 4, an individual is the entity which isbeing evaluated. The selected portions are each evaluated to determineif they potentially represent an individual and are assigned an initialconfidence value, as shown at block 410. If the selected portion has aninitial confidence value greater than a threshold, then the selectedportion is evaluated with respect to contextual relationships with otherselected portions, as shown at block 420. Next, as shown at block 430,based upon the contextual relationships, a context graph is built and acontext confidence value is established for each selected portion. Asshown at block 440, the selected portion's initial confidence value isupdated with the context confidence value and instructions are providedaccordingly. For instance, if the scenario involves an autonomousvehicle, then the instructions may require that the vehicle stop, changepaths, slow down, or continue on. If the scenario involves a securitysurveillance setting, then the future actions may include furthersurveillance, the manipulation of a camera or the dispatching ofofficials to the area.

As shown in FIG. 5, the method of developing a context graph 500 isillustrated. For each selected portion, a contextual relationship isundertaken with other selected portions for determining a contextconfidence value. In the exemplary embodiment, the context confidencevalue is developed utilizing four distinct contexts. These contexts areas follows: Spatial, Scale, Social and Overlap. Each context has anattraction component and Spatial, Scale and Overlap have a repulsioncomponent. As illustrated in FIG. 5, the Spatial Context Attraction isdetermined at step 510, and a Spatial Context Repulsion is determined atstep 520. The Scale Context Attraction is determined at step 530 and theScale Context Repulsion is determined at step 540. The Social ContextAttraction is determined at step 550. The Overlap Context Attraction isdetermined at Step 560. The Overlap Context Repulsion is determined atStep 570. These steps are repeated for a particular selected portionwith respect to other selected portions previously determined to exist.From the plurality of respective attractions and repulsions a contextconfidence value is created at step 580.

FIG. 6 illustrates the social context. Each circle represents oneselected portion. The arrows illustrate the body or head orientations.Two kinds of social patterns are modeled, following or facing, which isindicated by their respective body or head orientations. As shown inFIG. 6, the orientation of the head of an identified object 700 iscompared in relationship with the head of an identified object 710. Dueto the orientation, these entities are considered “following” eachother. The orientation of the head of an identified object 720 iscompared in relationship with the head of an identified object 730. Dueto orientation, these entities are considered “facing” each other. Therelationship of selected portions 740 and 750 is designated as followingas their orientation has them looking at the same area.

In certain circumstances FIG. 7 illustrates the label propagation withrespect to the contexts of individuals. As shown in FIG. 7, severaliterations of determining the respective context attractions andrepulsions are conducted in order to build the context graph. In theexemplary embodiment provided, as shown in FIG. 7a , an initial image isprocessed a first time to depict several selected portions 710-780. Ofthese selected portions, selected portion 720 is the clearest image ofan individual and receives a high initial confidence value that theobject is an individual. Utilizing 720 as a high confidence node, asecond iteration of the contexts is conducted in FIG. 7b . As shown inFIG. 7b , based on the contexts, selected portion 760 receives a highcontext confidence value and is also identified as an individual. Athird context iteration is conducted and with the high confidence ofindividuals 720 and 760, selected portion 730 receives a high contextconfidence factor and is identified as an individual. From theserespective individual nodes, the remaining selected portions 710, 740,750, 770 and 780 are also evaluated with respect to each other inpropagating a context graph. As shown in FIG. 8, the selected portions710 through 780 are each identified as an individual whereas many ofthese selected portions were not clearly representative of an individualfrom the initial image shown in FIG. 7 a.

As can be seen from the detailed description and accompanying drawings,one or more embodiments of the present disclosure provide technicaleffects and benefits, which include identifying with confidenceindividuals from initial selected portions by utilizing context basedpost processing techniques on opaque images which typically are notidentified utilizing standard image processing procedures. By utilizingpost processing techniques utilizing context based clues, moreindividuals may be identified in certain situations where previouslysuch individuals would be discounted. The increase in individualidentification has benefits in areas such as autonomous vehicleenvironments for detecting potential individuals in the path of thevehicle, and in crowd surveillance for identifying individuals who maybe attempting to hide their presence, or in situations where individualsexhibit unusually social behavior. Embodiment of the present disclosuremay be implemented as a computer readable storage medium. The computerreadable storage medium can be a tangible device that can retain andstore instructions for use by an instruction execution device. Thecomputer readable storage medium may be, for example, but is not limitedto, an electronic storage device, a magnetic storage device, an opticalstorage device, an electromagnetic storage device, a semiconductorstorage device, or any suitable combination of the foregoing. Anon-exhaustive list of more specific examples of the computer readablestorage medium includes the following: a portable computer diskette, ahard disk, a random access memory (RAM), a read-only memory (ROM), anerasable programmable read-only memory (EPROM or Flash memory), a staticrandom access memory (SRAM), a portable compact disc read-only memory(CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk,a mechanically encoded device such as punch-cards or raised structuresin a groove having instructions recorded thereon, and any suitablecombination of the foregoing. A computer readable storage medium, asused herein, is not to be construed as being transitory signals per se,such as radio waves or other freely propagating electromagnetic waves,electromagnetic waves propagating through a waveguide or othertransmission media (e.g., light pulses passing through a fiber-opticcable), or electrical signals transmitted through a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

What is claimed is:
 1. A computer implemented method for detecting oneor more entities in a scene, the method comprising: receiving, using aprocessor system, image data representing objects of a predeterminedarea; making an initial determination, using the processor system, thatselected portions of the image data represent the one or more entitiesof a selected class; determining an initial confidence value of theselected portions of the image data, wherein the initial confidencevalue identifies a likelihood that the selected portions of the imagedata represent the one or more entities of the selected class;determining a context confidence value of the selected portions of theimage data, wherein the context confidence value identifies a contextualrelationship between the selected portions of the image data; andutilizing said context confidence value and said initial confidencevalue to determine that one of the selected portions of the image datarepresents a first entity of the selected class, wherein said step ofmaking an initial determination that selected portions of the image datarepresent the first entity of the selected class includes utilizingfacial recognition attributes and assigning a confidence value basedupon said facial recognition attributes that the first entity is anindividual.
 2. The computer implemented method of claim 1, wherein saidinitial confidence value includes a threshold value being assignedcorrelating to the selected portions of the image data representing oneor more selected entities and wherein if said initial confidence valueis beneath said threshold value said selected portion is deemed notbeing the first entity of the selected class and is not included in theevaluation of other selected portions of the image data.
 3. The computerimplemented method of claim 1 wherein a selected portion of image datais compared to another selected portion of image data via a scalecontext and said scale context compares the height of a first identifiedobject with a second identified object.
 4. The computer implementedmethod of claim 1 wherein a selected portion of image data is comparedto another selected portion of image data via a spatial context whereinthe spatial distance between two selected portions selected portions iscalculated.
 5. The computer implemented method of claim 1 wherein aselected portion of image data is compared to another selected portionof image data via a social context wherein the social context measuresthe relationship of pose and body orientations to determine if aselected portion of image data is facing or following another selectedportion of image data.
 6. The computer implemented method of claim 1wherein an image boundary of a selected portion is compared to an imageboundary of an adjacent selected portion for determining if an overlapcontext exists such that the two selected portions are determined to berepresentative of the same individual.
 7. The computer implementedmethod of claim 1 wherein an identified object is compared to anotheridentified object via a scale context and said scale context comparesthe height of a first identified object with a second identified object,via a spatial context wherein the spatial distance between twoidentified object is calculated, via a social context wherein the socialcontext measures the relationship of pose and body orientations todetermine if a identified object is facing or following anotheridentified object and providing an aggregate context confidence value.8. The computer implemented method of claim 7 wherein said spatial,scale, and social contexts include an attraction component and saidspatial and scale contexts include a repulsion component.
 9. Thecomputer implemented method of claim 8 wherein a selected portion of animage is deemed not to be an individual if said aggregate contextconfidence value is less than a threshold.
 10. The computer implementedmethod of claim 8 further including moving a camera to capture anadditional image based upon the detection of an individual at an outerboundary of said initial image capture.
 11. The computer implementedmethod of claim 10 further including moving a camera to zoom into aspecific selected portion of the image if a respective context indicatesunusual behavior.
 12. A computer program product for detecting one ormore entities in a scene, the computer program product comprising acomputer readable storage medium having program instructions embodiedtherewith, wherein the computer readable storage medium is not atransitory signal per se, the program instructions readable by aprocessing circuit to cause the processing circuit to execute: receivingimage data representing objects of a predetermined area; making aninitial determination, that selected portions of the image datarepresent the one or more entities of a selected class; determining aninitial confidence value of selected portions of the image data, whereinthe initial confidence value identifies a likelihood that the selectedportions of the image data represent the one or more entities of theselected class; determining a context confidence value of the selectedportions of the image data, wherein the context confidence valueidentifies a contextual relationship between the selected portions ofthe image data; utilizing said context confidence value and said initialconfidence value to determine that one of the selected portions of theimage data represents a first entity of the selected class, wherein saidstep of making an initial determination that selected portions of theimage data represent the first entity of the selected class includesutilizing facial recognition attributes and assigning a confidence valuebased upon said facial recognition attributes that the first entity isan individual.
 13. The computer program product of claim 12, wherein aselected portion of image data is compared to another selected portionof image data via a scale context and said scale context compares theheight of a first identified object with a second identified object, viaa spatial context wherein the spatial distance between two selectedportions is calculated, via a social context wherein the social contextmeasures the relationship of pose and body orientations to determine ifa selected portion is facing or following another selected portion andproviding an aggregate context confidence value.
 14. The computerprogram product of claim 12, wherein said spatial, scale, and socialcontexts include an attraction component and said spatial and scalecontexts include a repulsion component.
 15. The computer program productof claim 12, wherein a selected portion of image data is deemed not tobe an individual if said aggregate context confidence value is less thana threshold.
 16. An individual detection system for detecting one ormore individuals in a scene utilizing an image capturing device, thedetection system comprising: a processor in communication with one ormore types of memory, the processor configured to: receive an image datarepresenting objects of a predetermined area; making an initialdetermination that selected portions of the image data represent one ormore individuals; determining an initial confidence value of theselected portions of the image data, wherein the initial confidencevalue identifies a likelihood that the selected portions of the imagedata represents the one or more individuals; determining a contextconfidence value of the selected portions of the image data, wherein thecontext confidence value identifies a contextual relationship betweenthe selected portions of the image data; and utilizing said contextconfidence value and said initial confidence value to determine that oneof the selected portions of the image data represents a selectedportions individual, wherein said step of making an initialdetermination that selected portions of the image data represent thefirst entity of the selected class includes utilizing facial recognitionattributes and assigning a confidence value based upon said facialrecognition attributes that the first entity is an individual.
 17. Thedetection system of claim 16 wherein said processor is furtherconfigured to determine an overall context confidence value by comparinga identified object with another identified object via a scale contextand said scale context compares the height of a first identified objectwith a second identified object, via a spatial context wherein thespatial distance between two identified object is calculated, via asocial context wherein the social context measures the relationship ofpose and body orientations to determine if a identified object is facingor following another identified object and providing an aggregatecontext confidence value.
 18. The detection system of claim 17 whereinsaid processor is further configured to manipulate a camera depending onthe value of a certain context confidence value with respect to arespective selected portion.